Considerations when figuring out how to create a streaming app

The rise of live streaming applications has revolutionized the way we consume content, allowing people from all over the world to interact in real-time. The world of live streaming has expanded beyond just gaming and social media, with platforms like Zoom, Microsoft Teams, and Google Meet being used for remote work and online learning. Live streaming has also enabled remote fitness classes, concerts, sports events, and other activities that were previously limited to physical attendance. Live streaming applications are becoming increasingly popular, with users expecting high-quality content and a seamless experience. The success of these platforms largely depends on the ability to provide a reliable and consistent live streaming experience. This requires overcoming technical challenges, such as synchronization, quality, and latency issues, which are all relevant when figuring out how to develop a video streaming app.

The Challenge: Custom Uploaded Content and Local Playback

When You Concider to Create a Music Streaming App or Video Streaming App...

When an instructor uploaded custom video and music to his session, the content was played locally on the participant's device rather than being streamed in order to decrease network load and improve performance. However, when custom video and music were played locally, the latency between the instructor and participant could cause a delay in the playback of the content, leading to participants being out of sync with the instructor. This is a significant problem in fitness classes where precise timing is crucial, as participants may miss out on important cues and movements.

Moreover, the different network conditions and device capabilities of each participant may cause playback quality variations, leading to inconsistencies in the fitness experience. This can result in a lack of engagement and potentially lead to a negative user experience. Therefore, Moravio had to come up with a solution to synchronize the video and music playback for all participants.

There are a number of possible solutions to the synchronization issue in general:

Timecode synchronization With timecode synchronization, the instructor’s playback time information is forwarded to the participants in real time and their video and music playback is adjusted to equal the instructor’s playback time. This is the solution that Moravio has implemented originally, albeit with limited results. The problem is that network quality and device performance might still cause a significant delay in the playback compared to the instructor’s camera and microphone streamed tracks, again causing inconsistencies.

Delaying the stream Another solution is to delay the stream slightly to allow for any delays in the network. This means that the instructor would stream the content slightly later than usual to allow for the network to catch up. This method ensures that all participants receive the same content at the same time and reduces the chances of delays or out-of-sync content. However, this solution is unusable for this use case, because the instructor needs to react in real-time to any inputs from the participants, which would be slightly delayed, causing confusion. Keep in mind that things like these are important to consider when learning how to create a streaming app.

Streaming the locally played content separately Instead of letting the participants’ play the custom video and music locally, the video and music playing on the instructor’s side can be captured and streamed as separate tracks alongside the instructor’s camera and microphone tracks. This is the solution that Moravio ultimately implemented to solve the synchronization problem.

A Solution: Capturing and Streaming Custom Content

When asked about how to build a live video streaming web application and trying solve the synchronization problem, Moravio has implemented a solution where the custom video and music that was previously playing locally on both the instructor and participant side in an HTML media player is now playing locally only on the instructor side and is being captured and streamed as a separate video and audio track. This means that the participants are no longer playing the custom video and music locally, but instead, it is being streamed to them from the instructor's device. This ensures that the instructor’s workout cues are in sync with what is actually happening on the stream and all participants are following the fitness moves in real-time.

The process involved in capturing and streaming custom uploaded content as a separate track includes the following steps:

1. The instructor uploads the custom video and music to the session using the class builder.
2. We capture the custom video playing in the HTML media player using the captureStream() method when it is playing
3. We also capture the music using the createMediaStreamSource() method on the AudioContext
4. The captured video and audio are then streamed to the participants as separate tracks.

The benefits of this approach include a seamless and synchronized fitness experience for both the instructor and the participants. This creates a more enjoyable and effective workout session and ensures that participants are following the instructor's moves accurately. The downside to this approach is that because the network load is increased, participants with poor network connectivity may experience drops in video and music quality. However, LiveKit automatically takes care of adaptively changing the video quality based on the network connection quality, which is definitely key in scenarios like this one.

Furthermore, this solution highlights how technology can be used to improve the quality of live streaming applications and enhance the user experience. It is a great example of how innovative solutions can address challenging issues and ultimately lead to a better product for users. Hopefully this article will provide you with some tips on how to make a streaming app.