The arms race between AI-identifying software and AI software is potentially going to make it impossible to really "know" if the AI-identifying software is even working. I'm under the impression most developers or all developers do not understand exactly what is happening inside these machines during runtime / training time.

Most court systems are sluggish and highly bureaucratic. They aren't going to be able to keep up or decide what AI-identifying software to use when different systems could be rapidly coming out.

As others have said, media evidence could be accepted if it's verified as coming from a trusted source. Unfortunately this causes a sort of "who watches the watchers" scenario -- it's sad because for so long video evidence was basically an oracle of truth.