Droidkaigi 2019 (part 1)

This is the first part in my series about my talk at DroidKaigi 2019. You can find the rest of them below:

• Part 1 (this post)
• Part 2
• Part 3
• Part 4

0. Attending DroidKaigi 2019

One of my TODOs this year is to attend more tech conferences, starting from Android and will include other technologies as well. And not just being there listening, I would like to share what I'm doing too. In 2 days from Feb 7th, Feb 8th this year, I had a chance to be speaker at DroidKaigi 2019 in Tokyo Japan where I'm located in.

The overall experience was great. The event is so well organized thanks to the team. Though my talk was not so good, I could finish just 80% of it, so it is not quite a good start for me this year. To be honest, after my CFP was accepted, I got a bad health condition and need to spend couple weeks in the hospital. After going back, the remaining time was quite tight so it may be one of the reason. But after all, I know that I need to do much more to be better next time.

Ok. The remaining of this post, I would like to share my talk at DroidKaigi, including part that I could not finish.

1. My talk about 'ExoPlayer in RecyclerView'

The title is "ExoPlayer in RecyclerView, a proposal" and you can get the deck here:

The talk is structured as below:

• Brief introduction, about the talk and about the speaker.
• Motivation: why I want to talk about ExoPlayer in RecyclerView.
• Common approach to integrate ExoPlayer in RecyclerView and its issue from my point of view.
• My proposal to fix them and make the implementation better.
• I also share a demo App to demonstrate how my proposal can help to build various UX that involve ExoPlayer in not only RecyclerView but also ViewPager, NestedScrollView as well.

1.0 Before we start

I have built a demo application for this talk. The source code is on github as well. Before getting further, I would like to share links them here, so for those who has already go though my talk and my slides, you can grab the code right here.

Demo App (Google Play Beta) Source Code (library & sample)

1.1 About speaker and the motivation behind

I'm Nam and I have been using using 'eneim' (pronounced /ɛn/ /eɪ/ /ɛm/) for a while.

For those who are not familiar with ExoPlayer and RecyclerView yet, I suggest to go to ExoPlayer repository and RecyclerView documentation first.

The motivation behind the topic "ExoPlayer in RecyclerView" comes from recent applications like Facebook and Twitter, where their timeline contain various types of content including Video, and those Video will be played/paused on User interaction (scroll, swipe, etc). Furthermore, changing device orientation or opening the App in multi-windows mode will automatically open fullscreen or more rich UXes.

It is not just talking, but there is also actual need out there. Here I would like to mention a 3-year-old issue on ExoPlayer repository, which is not resolved yet today:

"How to use ExoPlayer in a ListView or RecyclerView?"

I want to use ExoPlayer in a RecyclerView as a part of row item.
I want to make a customer view and wrap the ExoPlayer in that view.
 
Do you have some advice?
 
Thank you!

After a few years working on this requirement, I understand that a full implementation would at least contain:

- Video player in RecyclerView, ViewPager, ScrollView, etc 
- ExoPlayer or whatever works
- Auto play/pause on scroll? (like Facebook, Instagram, Twitter, etc)
- Fullscreen back and forth, smoothly
- Network usage friendly, battery friendly, UX/UI friendly, etc friendly

To understand more on this requirement, I investigated some top of the class applications, figure out how they are doing.

1.2 Market research

1. Facebook

Facebook app has the following good UX implemented:

- Auto play/pause on scroll
- Dialog for fullscreen player
- Auto fullscreen on landscape

But what is not so good from my observation is: the playback is unstable under config changes, video reload on config change?.

2. YouTube

YouTube is another really good app. It has rich UI/UX and it was really wel-built. The playback UX is as below:

- Great UX in single item play
  - Has mini/overlay player when user swipe down.
- Well config change handling
  - Animation on layout change, etc.
- Latest: optional auto play in list

Again, there is a tiny thing I concern about: it would reload the Video when I switch from list to single player.

3. AbemaTV

Last but not least, AbemaTV app, one of my favourite TV app in Japan. Beside good TV playback experience, it has

- ViewPager + ExoPlayer?
- Auto switching to fullscreen on landscape and back

AbemaTV shows a channel in a Page of ViewPager, and when I change a page, then back to it right after, it will reload the channel, which is not so great.

Here, I would like to emphasize on what is my goal and why I always concern about the reloading issue: the playback continuity.

What is that?

Playback continuity is an important part of Video playback experience. So when playing a Video, you have the image part, and the sound part. In Android, the image part is rendered to Surface object, and the sound part will be 'rendered' to the standard output. The playback continuity is the experience of having both, or one of them, being rendered continuously during the timeline.

Why 'both, or one of them'? In Android, the whole playback experience would include the switching device orientation for larger visible area, changing window size to make room for other app, or opening Picture-in-Picture mode to serve special case. Each of them will trigger the well-known configuration change in Android system. If you handle the config change by hand (by adding a manifest entry to Activity, and overriding onConfigurationChange method), you are able to keep both of the streams. But if you have the system handle it for you (by not having manifest entry), because the UI will be recreated, your Surface object will also be recreated as well. (In fact, at really low level, I believe one would retain the Surface object (not the SurfaceView or TextureView instance) for reusability, but it requires much more effort. I will investigate further on this, but in this talk, I consider it is out of my concern). This recreation will discontinue the image stream. Therefor, your best effort is to have the sound part keep playing during the configuration change.

My proposal in this talk focus on keeping the actual playback continuity, by having the following behavior:

- No resource reloading/rebuffering.
- No audio discontinuity (accept the case of buffering 
next piece of resource).
- Video discontinuity only on Surface recreation.

1.3 Challenges, common approach and the not so good part

1. Challenges

What would be challenges when making ExoPlayer to work in RecyclerView? In fact, it is not too challenging if once could keep in mind some certain criteria like performance issues, reclycing of the ViewHolder, etc. But to have the playback continuity with decent config change handling, I find it to be really challenging.

To work with RecyclerView, you should be aware of the following points:

- RecyclerView: scrollable, infinitely
- Item (ViewHolder) is supposed to be reused/recycled 
- ViewHolder lifecycle
  - Created/Bound/Recycled
  - Attached/Detached

And to work with ExoPlayer in RecyclerView, the points below are also important:

- Many player instances = system performance down
  - The more player instances, the lower performance
- Many PlayerView = many Surfaces = surface creation/management cost
  - The more Surfaces, the worse

Furthermore, if your app also allow user to open a single player in fullscreen, and back, there are more problems we need to think about:

- UI flow: decide how the UI of fullscreen would be: in different Activity? 
Replacing current Fragment? Multi-Windows mode?
- Chances for configuration changes
  - Config changes handling is complicated
  - Config changes will impact playback continuity
- Not only orientation change

Most of the approaches use single ExoPlayer instance and reuse it across multiple playback. But I find that controlling them too strictly, it is the UX to be affected. An example is the case of AbemaTV app above.

2. Common approach & Issues

Following the discussion on this issue on ExoPlayer, as well as other approaches on github, I can summarize a common approach to have ExoPlayer work with RecyclerView as below:

- PlayerView in ViewHolder

- Adapter: manage ExoPlayer and MediaSource, 
  - MediaSource is created and bound to VH on demand
  - Only “will play” ViewHolder will be provided by the ExoPlayer instance

- Adapter’s callback to update playbacks
  - onBindViewHolder → set MediaSource
  - onViewAttachedToWindow/onViewDetachedFromWindow → prepare/release?
  - onViewRecycled/onFailedToRecycleView → TODO?

- RecyclerView callback to update playbacks (OnScrollListener)
- Playback strategy: top-most visible → play, otherwise → pause
- Reuse PlayerView: remove from “will pause” ViewHolder then 
add it to “will play” ViewHolder

This approach is well thought, and could be enough to solve those challenges above. But it lack of the way to handle fullscreen back and forth, as well as it has tight control over playback resource, which is good for performance, but user experience will be impacted.

I summarize what could be the issues of this approach below:

- Efficient Player management = UX loss
  - Reuse one Player for many PlayerView → “black splash” on resume (like AbemaTV app)
  - Re-buffering takes time
- ViewHolder lifecycle
  - onViewDetachedFromWindow is not always called
  - setAdapter(null) ← to do or not to do?
- UI Flow: to fullscreen and back
  - Chance for configuration changes

Last but not least, if an app allow 'Video reloading on config change', there is one last concern I would like to talk about, which I call "the hidden issue of Video reloading".

From the business point of view, playback view time is an important metrics. I cannot say for all applications, but how this view time is counted depends on both client and server side. Says 'User views more than 50% of a Video length = 1 view time counted'. It is easy to observe if user has seen more than 50% length of a Video. But what happens when that user rotate the device to view in larger area? If this triggers a reloading, will it be counted as another view time by server? Or how would you handle that change?

Of course, it is the policy you define, and the mechanism you track the view count. But not reloading the Video will save you a lot of headache handling it.

1.4 The proposal

We have discussed about the motivation, the challenges, the common approach and its issues. From now I would like to introduce my approach and how it can not only solve many problems, but also be helpful to build many different user experiences, not just with RecyclerView.