Interesting! it had crossed my mind initially that maybe there was never inlay in the area, as you say, it is quite flat. The front side of the Tsuba has been quite corroded , especially on that right hand side, so I figured that maybe some of the corrosion has taken away not only the inlay, but also the carving definition?
some inlay on the front lower right is loose which I am almost sure is due to the old corrosion too.
I feel like I have seen this design in a book from Gary D Murtha in the past, long before I received this Tsuba. I might have to find a copy of them and have another look!