The video buffer took ownership of the producer frame (so that it could
swap frames quickly).
In order to support multiple sinks plugged to the decoder, the decoded
frame must not be consumed by the display video buffer.
Therefore, move the producer and consumer frames out of the video
buffer, and use FFmpeg AVFrame refcounting to share ownership while
avoiding copies.
This flag forced the decoder to wait for the previous frame to be
consumed by the display.
It was initially implemented as a compilation flag for testing, not
intended to be exposed at runtime. But to remove ifdefs and to allow
users to test this flag easily, it had finally been exposed by commit
ebccb9f6cc.
In practice, it turned out to be useless: it had no practical impact,
and it did not solve or mitigate any performance issues causing frame
skipping.
But that added some complexity to the codebase: it required an
additional condition variable, and made video buffer calls possibly
blocking, which in turn required code to interrupt it on exit.
To prepare support for multiple sinks plugged to the decoder (display
and v4l2 for example), the blocking call used for pacing the decoder
output becomes unacceptable, so just remove this useless "feature".
A skipped frame is detected when the producer offers a frame while the
current pending frame has not been consumed.
However, the producer (in practice the decoder) is not interested in the
fact that a frame has been skipped, only the consumer (the renderer) is.
Therefore, notify frame skip via a consumer callback. This allows to
manage the skipped and rendered frames count at the same place, and
remove fps_counter from decoder.
Limit source code to 80 chars, and declare functions return type and
modifiers on a separate line.
This allows to avoid very long lines, and all function names are
aligned.
(We do this on VLC, and I like it.)
The decoder initially read from the socket, decoded the video and sent
the decoded frames to the screen:
+---------+ +----------+
socket ---> | decoder | ---> | screen |
+---------+ +----------+
The design was simple, but the decoder had several responsabilities.
Then we added the recording feature, so we added a recorder, which
reused the packets received from the socket managed by the decoder:
+----------+
---> | screen |
+---------+ / +----------+
socket ---> | decoder | ----
+---------+ \ +----------+
---> | recorder |
+----------+
This lack of separation of concerns now have concrete implications: we
could not (properly) disable the decoder/display to only record the
video.
Therefore, split the decoder to extract the stream:
+----------+ +----------+
---> | decoder | ---> | screen |
+---------+ / +----------+ +----------+
socket ---> | stream | ----
+---------+ \ +----------+
---> | recorder |
+----------+
This will allow to record the stream without decoding the video.
Several frames may be read by read_packet() before they are consumed
(returned by av_read_frame()), so we need to store the PTS of frames in
order, so that the right PTS is assigned to the right frame.
Since PTS handling has been fixed, the recorder do not associate a PTS
to a wrong frame anymore, so PTS of "configuration packets" (which never
produce a frame), are never read by the recorder. Therefore, there is no
need to ignore them explicitly, so we can remove the MediaCodec flags
completely.
The PTS was read from the socket and set as the current one even before
the frame was consumed, so it could be assigned to the previous frame
"in advance".
Store the PTS for the current frame and the last PTS read from the
packet header of the next frame in separate fields.
As a side-effect, this fixes the warning on quit:
> Application provided invalid, non monotonically increasing dts to
> muxer in stream 0: 17164020 >= 17164020
SDL_net is not very suitable for scrcpy.
For example, SDLNet_TCP_Accept() is non-blocking, so we have to wrap it
by calling many SDL_Net-specific functions to make it blocking.
But above all, SDLNet_TCP_Open() is a server socket only when no IP is
provided; otherwise, it's a client socket. Therefore, it is not possible
to create a server socket bound to localhost, so it accepts connections
from anywhere.
This is a problem for scrcpy, because on start, the application listens
for nearly 1 second until it accepts the first connection, supposedly
from the device. If someone on the local network manages to connect to
the server socket first, then they can stream arbitrary H.264 video.
This may be troublesome, for example during a public presentation ;-)
Provide our own simplified API (net.h) instead, implemented for the
different platforms.
The skip_frames flag was a non-configurable runtime flag. Since it is
not exposed to the user, there is no need for a (possible) runtime cost.
For testing purpose, we still want it to be configurable, so make it a
compilation flag.
Romain Vimont
Bernhard Rosenkränzer
Marco Martinelli
Ivan Gorinov
Romain Vimont