ESP32 game console/phone/media player with TFT and PAL/NTSC AV output via VS23S010/040

[RetroZvoc]

Hello.
I'm a computer enthusiast from Croatia studying electronic business at Zagreb University of Applied Sciences. My field of interest is consuming and developing video games, software, music and art, and I would love to utilize that for my future enterpreneurship projects for Millennial nostalgia; specifically Windows XP era, Sony Ericsson featurephone era and SNES/GBA/PS1 game console era (1995-2005).

I'm trying to make my own multimedia device akin to the Sony Ericsson K510i/K610i/C510/W705 featurephones which had so much multimedia like Java apps (Opera Mini for web browsing, VibeJive for music composition, PaintCAD for pixelart, eBuddy for instant messaging for MSN/Facebook/ICQ/etc., MiniCommander for media reproduction, VideoDJ for video editing, etc.). Those Sony Ericssons don't exist anymore and Android phones have very bad memory management and don't provide good security or certainty; whether it's due to how you cannot pick up the phone without accidentally pressing something or due to how much privacy is being violated and many features being far more glitchy. I would love to make an embedded device that could be a phone, a media player, a game console or a portable computer, whether it's for home use or portable.

There have been many attempts at making a game console or a media device that can send a full frame without lag. I've had experience with ATMEGA328P and ATXMEGA128A1U, but despite the SPI, 8-bit bus bitbanging, external bus interface, etc., it was still impossible to render a full image while having some spare power to do other things because the CPU would either be bitbanging too much, the SPI would be slow, or the DMA would halt the CPU. ESP32 seemed like a good solution, but despite its 240MHz dual-core DSP CPU, the SPI is limited to 80MHz and the display I'm using is max 40MHz. And my calculations say that it's impossible to have a full frame of 320x240@60fps.

However, the ESP32 has a hardware Quad-SPI functionality with DMA-powered FIFO buffer which I could utilize if only I had a display with a Quad-SPI interface. Having had experience with a Geeetech board of VS1053 and soldering unused GPIO pins for Real-Time MIDI and having almost broken it by bad wiring (it produced creepy sounds) and then repairing it somehow, I remembered how awesome your products were/are and I decided to take a little look and I saw that you have an amazing video chip!

That's where I found your VS23S040 chip. It seems like an excellent solution! I would just like to know what features could be possible.
ESP32 has 520kB of internal RAM, external serial flash depending on development board, possible external serial RAM, and 240MHz of a dual-core DSP CPU. It's used for WiFi and Bluetooth and many IoT applications. My application currently is a game console/media player. There are also two more hardware SPI interfaces that can run up to 80MHz and have a DMA controlling them; they can be single wire, two wire, or four wire SPI (Quad-SPI).

If my calculations are correct, then if ESP32 could send a scanline buffer's 24-bit picture via Quad-SPI at 40MHz (I wish it could be 80MHz because ESP32 can do it) for a 320x240 resolution at 60FPS without halting the CPU during the DMA transactions, the used bandwidth would be 69.12% which is awesome! Alternatively, 50FPS could make it 57.6%. I already have the graphics engine in mind with prepared tilesets and nametables akin to NES/SNES as well as HDMA scrolling/bankswitching and other fancy effects which could easily render onto a new scanline buffer while the old one is being sent.

I would like to see if anyone has attempted to do this and where I could get a development board of VS23S040 and if that chip could be used as the serial RAM chip so that the ESP32 can map it onto its own address bus; or if that would halt the CPU, just use it as yet another SPI device.

Another feature I would like is one of your VS1000 chips to provide these features:
1. USB-to-USART interface (akin to Arduino's FTDI/CH chip) for programming ESP32.
2. Access to SD card, NAND Flash chip and another USB device so that both the VS and ESP can use those memories simultaneously.
3. USB Host so that the end users can use USB devices like keyboards, mice, joysticks and USB HID Serial devices so that my ESP32 can program another ESP32 (yay for selfsustainable computing! )
4. Encoding, decoding and mixing WAV, OGG, Vorbis, Opus, MIDI (both as a file and real-time) and raw audio from my ESP32's I2S audio, on-board microphone and Line-In microphone which could all serve as background music in video games, sound effects on video games and user interfaces, media playing, audio processing for audio editing programs, audio recording, and possibly online game streaming. ESP32 is capable of producing and recording audio, so some of the processing power might be used by ESP32 if it's a stretch for VS. For example, VS could be playing real-time MIDI commands from my ESP32's commands while playing OGG voice clips for character dialogue and WAV for sound effects and the ESP32 could be playing in-game player voice clips recording the whole gameplay and streaming it to Twitch.
5. It has to be compliant with MIT license or Apache License 2.0 which I would love to use in my projects in order to have it Open Source / Open Hardware certified. This means there shouldn't be any MP3 or WMA patented/copyrighted material unless it is possible for the user to alternatively download those for non-commercial use.
6. Video processing and picture processing of free and open-source (MIT/BSD/Apache) codecs (Theora, PNG, BMP, JPG and possibly a custom format which I could program that could send a bit of JPG and a bit of OGG simultaneously). This should be possible to make my device a media player as well as a media recorder in case a camera is used and a media streamer for Twitch gaming Let's Plays.
Which VS chip could do this? How many of them could there be?

How much would all that cost to use with my NodeMCU? Do you have any development boards of these? How much would it cost for you to make a custom PCB with these components for me either physically or digitally so that a PCB assembly company in my country could do it to reduce shipping expenses since Croatia has recently got huge shipping issues?

I bet you've never seen anything as long as this before , but I grew up on forums and learned to write long essays for school and to hold very specific presentations for college.
So thank you for your time to read through all of this.

Best regards,
(Codename) RetroZvoc

[Panu]

Hi!

You have some great ideas! I'm not at the office, but here's a small reply with some ideas.

You should get a VS1010 Developer Board to start with. It contains a VS23S010 with composite video output. There's a PAL video player demo for that board. Also I*ve done a blitter/tilescroller demo with it - you do know there's a blitter inside the VS23S010 that can copy square areas from one place to another.

I like the 256 color modes best, since it gives a nice 1 byte per pixel memory layout, and you can keep a nice image with a full framebuffer without any need to constantly update the video memory. That said, if you do want to update the whole video memory all the time, e.g. "chase the beam", I think it is possible with careful bus timing, but there's not much time to spare. The 8 bit parallel bus would be the best for that, I think. It all depends on the resolution and number of colors.

There's all sorts of great little tricks you can do with the hardware, such as update line pointers in real time, duplicate lines, or change the color depth at real time for different lines, and use the blitter, to get the most of the chip.

As for your other ideas, I'd look into the VS1010 for those as well.

Keep up your enthusiasm and ideas! Let's keep in touch!

Panu

[RetroZvoc]

Thank you for your response!

My idea as always was to use the fastest DMA possible to transfer video data from scanline buffers on the main microcontroller onto which I'd manually render the nametables, tilesets, 2D polygons made out of 3D matrix projections, sprites, phone UI bitmaps, fonts, etc..

For the ESP32 which has 240MHz of a dual-core DSP CPU, the maximum IO is a QuadSPI at 80MHz and DMA. Before I came here, I thought that a great idea would be to use an FPGA which could store 4-by-4-bits into a byte and then send those as a parallel 8-bit 40MHz data transfer.

However, that wouldn't be efficient for interfacing a modern display since modern displays have 16-bit colors and because calculating them and merging them is a mathematical issue which requires a lot of CPU power.

Your VS23S040 chip solves this easily because every pixel is 8-bit and a YUV pallette example (somewhere on this forum) shows that the lower 4 bits can be used for shading curved 3D objects where lighting and shading is used and the higher 4 bits can be used for color hue selection. I just don't know could this be as fast as I expected it to be. Could the VS chip allow for a 80MHz QuadSPI and could that QuadSPI-to-8-bit-parallel FPGA solution of mine be a good idea if the VS chip isn't able to handle the 80MHz? What if I use an LCD screen in combination with an AV PAL output so that my console can be used as a home/portable hybrid like Nintendo Switch?

My approach wouldn't be to chase the buffer, but to do a dual-buffering. The phone's GUI (status bar on top and bottom softkeys) would be prerendered and there wouldn't be any need to rerender it every single time (unless it's animated). The dual-buffering would be done in a way that if there's a lag in the game, the old frame is shown on the current buffer while the new one is being rendered in the background one. When a new frame has finished rendering or is in the process of being rendered without a probability of tearing, the frame-buffer pointers would swap and that way the previously-old frame would now become the currently frame. I think that that's how PS1 handled Driver 1/2 graphics as the game was never able to keep up to 50/60fps.

If a retro game is being played such as an NES homebrew game or a Pico-8 game or something with a tile-based graphics game engine, I would love to see if that blockcopy mechanism in your VS chip could render both the background tilesets and the transparency sprites at the same time or if I need to do that manually. That way retro games would consume less CPU power and have better battery life which would be a great marketing trick. "More retro games, more green environment. "

Since I'm a student with not so much money and since shipping to Croatia is a little problem, I would like you to recommend to me which one of your solutions or some Adafruit/Sparkfun would be the best for me to order from a PCB maker in Europe.