Describes the feature set and how to work with the Virtual App
This App is used in conjunction with the Virtual Plugin to transmit and receive data over IP between the device and PC. The only user input required in the app is the IP address of the PC.
Streaming Virtual Reality Game to Android mobile
Remote Session App
Broadcast to the PC position and
rotation of camera or object
This technology allows you to turn Android devices into a virtual joystick which will be useful for designers and developers who create new game mechanics and work with models.
Game mechanic in Unreal Engine 4
The Virtual App enables a user to drive a Camera and Objects using an Android device in a virtual production environment. With ARCore the position and rotation of the Android devise is broadcast wirelessly to the PC.
This technology allows you to turn Android devices into a virtual camera, which will be useful for designers and developers who create new game mechanics and work with game cameras.
Virtual production in Unreal Engine 4
Virtual Plugin for Unreal Engine 4
Remote Session Plugin
Video striming from PC
Broadcast to the PC position and
rotation of camera or object
The plugin allows to use any objects in the scene, including all blueprints, cameras, etc. and move them in real-time in the game scene using the Virtual App, which transfers the movement and rotation data of an Android device in space.
How to start work with UE4 plugin
How to configure virtual camera to work with nDisplay
- Unreal Engine 4.23
- Windows 10, 64-bit
- GeForce GTX 1060
- Intel i5 7th Generation CPU
- 8 GB RAM
- ARCore-capable Android device for Virtual App
- Wi-Fi physical layer speed according to IEEE 802.11n > 200 Mbps
- GeForce GTX 1070 or above
- Intel i7 7th Generation CPU or above
- 16 GB RAM or above
- Wi-Fi physical layer speed according to IEEE 802.11n > 300 Mbps
1. Run your project, restart it if necessary. When the project has successfully started go to Edit-> Plugins-> Other and make sure the plugin is active.
If you added plugin from the marketplace, you should go to “Content Browser” and select “Show Engine Content”.
For new users! Open “Camera Content” in “Content Browser” and drag “MyActor” to your scene.
For existing users! Open your “Actor blueprint” and find in “Details” section “BPClasses”. Select “WBP_MainUserInterface”.
2. Open “Project settings” and find “Input” section, select “Always Show Touch Interface”.
Now you will have access to the interface during the launch of the scene with the settings of the joystick and video recording button.
In “MyActor” you can faind “Multiplication by number” section in wich you can control the scale of the movement of the camera/object by increasing or decreasing the multiplier, the value of “1000” is 1 : 1.
In the “Multiplication by number” section you can control your coordinates “Movement” and “Rotation”. Also you can turn on and turn off all coordinats if you need.
In the “Body” section you can find “Auto Focus”. Also you can turn on and turn it off.
In the stabilization section, you can change the stabilization strength. For sensitive camera operation, select 0.5 and for very smooth operation, select 1.
In the “ARPhone/Bode” section you can control “Camera Follow” – mark it if you want to work wih virtual camera and unmark if yoy want to work with virtual joystick, mark and unmark “Debug Message” and choose a camera to display, it can be “Cinematic Camera” (you should add it to the scene before choosing).
In the “Component”, select the “Mesh”.
Add a new “Component” in your Blueprint.
Add a new “Blueprint” to the scene. Now it will be working in the editor and compiled project on Windows and Android platforms. Also, you can add a couple of same “Blueprints” in your game, our plugin is support multi-devices.
3. Now you can click the “Play New Editor Window” button in your scene. After that launch Virtual App and click Auto IP in app and run Virtual Camera Connect button.
Solutions for some popular problems while using Plugin
1. If you are sure, that the network, PC and mobiles are working well but you still have problems – this may help you:
Windows panel -> Settings -> Network&Internet -> Wi-Fi -> Windows Firewall and turn the firewall off, sometimes it can block data transfer on port 7777
2. If video streaming does not cover all screen on the phone or instead It cuts some part of the viewport – change the play mode to “New Editor Window” on the top toolbar or try to adjust the viewport size.
3. If you have a connection issue use not loaded access point to fix the problem, the problem can be if you work in a place with a lot of devices on one router.
4. If you want to use VR mode – enable “google VR plugin” and disable “steam VR” because it can block video streaming.
5. Turn off “RTX” when you use “VR mode” if you will not do it you will get problems with screen to render and your phone will have one black window in one of the VR sides.
6. If you want to build your project you need to turn off UI (MyActor->Details panel->User Interface->”Enable GUI”) in “MyActor”, otherwise you will get an error launching your project’s Exe.
Virtual Plugin UI
This section outlines the Virtual Plugin UI and its settings.
1 – Visibility
Tapping the Hide icon in the upper-left corner will hide/show the Virtual Camera UI.
2 – Focus Mode
The two buttons in the left corner of the UI ( A, M ) are used to set the Focus Mode for the Virtual Camera.
Auto (A) – The default setting. The focus distance will continually update to keep whatever is in the middle in focus.
Manual (M) – The setting which will set the focus distance you can change by your self. You can set a debug plane for better work with constant value each you can choose on the screen for changing focus distance.
3. Default Joysticks
You can click on a button and add joysticks for better camera control. The left joystick is responsible for moving forward and backward, and the right one for moving up and down. This is required for working with large locations. In the main menu, you can change the movement speed multiplier for each joystick separately.
4. Recording Button
Clicking the Record button initiates a new recording while clicking again stops the recording.
5. Focus Distance
The Focus Distance are displayed in the top-right region of the UI.
Tapping on the Gear icon in the upper-right corner will bring up the Settings menu.
The General Settings tab contains the ability to configure camera stabilization, multiplication of the camera movement, and multiplication movement of the right and left joystick separately.
Axis locking will allow you to lock any quantity of the axes. It can be handy for a linear dolly track effect or other cinematic tricks.
FilmBack settings will allow you to choose from a list of predetermined Film Formats and change Smooth Lens settings.
Pressing the Stop Tracking button will temporarily disable the input source so the view on the phone pauses, enabling the user to move the phone somewhere else in the real world. Pressing Continue Tracking then re-enables the input source but keeps the view offset.
You can add several cinematic cameras to the scene, pre-install them in the desired positions, and adjust all the settings for yourself. After that, just add them to the array so you can change them using the UI in the multi-camera section.
In the multi-camera section, you can select a pre-installed cinematic camera. After selecting a camera, you only need to click on the Unlock All Axes button. All preset settings will be displayed in the UI.
How to set remote video streaming
Prepare Your Unreal Engine Application
In this step, you’ll create a standalone executable file for your Project.
1. Open your Project in the Unreal Editor.
2. From the main menu in the Unreal Editor, select Edit > Plugins.
3. Under the Graphics category, find the Pixel Streaming Plugin and check its Enabled box.
4. Click Yes to confirm.
5. Click Restart Now to restart your Project and apply the change.
6. From the main menu, choose Edit > Editor Preferences…
7. For UE 4.23:
Under the Level Editor > Play category, find the Additional Launch Parameters setting, and set its value to -AudioMixer
For UE 4.24:
Under the Level Editor > Play category, find the Additional Launch Parameters setting, and set its value to -AudioMixer -PixelStreamingIP=localhost -PixelStreamingPort=8888
8. Now it is set to work in Standalone Game, if you want to set it for work at compiled project follow the next steps. (look at “Start the Servers ” stage to check how to set server)
9. Package your Project for Windows. From the main menu in the Unreal Editor, choose Files > Package Project > Windows > Windows (64-bit).
10. Browse to the folder on your computer where you want the Unreal Editor to place the packaged version of your Project, and click Select Folder.
11. The Unreal Editor begins the packaging process.
12. When the packaging process is finished, go to the folder that you selected. You’ll find a folder called WindowsNoEditor with contents similar to the following:
13. Every time you start your packaged application, you need to pass it the same command-line flags set in step 6 above. One way to do this is to set up a shortcut:
Press Alt and drag your .exe file to create a new shortcut in the same folder (or anywhere else you like on your computer).
Right-click the shortcut and choose Properties from the contextual menu.
For UE 4.23:
On the Shortcut tab of the Shortcut Properties window, append the text -AudioMixer -PixelStreamingIP=127.0.0.1 -PixelStreamingPort=8124 at the end of the Target field, and click OK.
For UE 4.24:
On the Shortcut tab of the Shortcut Properties window, append the text -AudioMixer -PixelStreamingIP=localhost -PixelStreamingPort=8888 at the end of the Target field, and click OK.
You now have a packaged, standalone Unreal Engine application that has the Pixel Streaming Plugin enabled, ready to stream its rendered frames and audio.
Start the Servers
In this step, you’ll start the web services that will help to establish peer-to-peer connections between your Unreal Engine application and clients’ browsers.
2. Only for UE 4.23
In your Unreal Engine installation folder, find the location of the Signaling Server under Engine/Source/Programs/PixelStreaming/WebRTCProxy/bin/WebRTCProxy
3. Only for UE 4.23
In your Unreal Engine installation folder, find the location of the file “Package.json” and change “bcrypt 2.0.1” to “bcrypt 4.0.1”
4. In your Unreal Engine installation folder, find the location of the Signaling Server under Engine/Source/Programs/PixelStreaming/WebServers/SignallingWebServer
5. Start the Signaling Server by running the run.bat file. The first time you run the server, it will download all the dependencies it needs. When the server has started and is ready to accept connections, you’ll see the following lines in the console window:
When the Unreal Engine application connects to the Signaling and Web Server, you should see the following line of output in the console window opened by the Signaling and Web Server:
Streamer connected: 1
This means that you now have the Unreal Engine application running with the Pixel Streaming Plugin enabled, and the front-end Signaling and Web Server is ready to route connecting clients to the Unreal Engine application.
If necessary, you can stop and restart the Unreal Engine application and the Signaling and Web Server independently. As long as they’re both running at the same time, they should be able to reconnect automatically.
At this point, you have everything you need set up and working on your computer and mobile phone. All that’s left is to connect a browser to test video streaming.
1. Open your Virtual App and add click auto IP and mark Remote.
2. You will find streaming session, press play.
3. Streaming is ready to work.