"Semidirect product" of graphs?

Well, the fact that you haven't gotten an affirmative answer despite this question receiving many votes and views, is your meta-answer right there. You must assume that NO, "semi-direct product" for graphs has NOT been defined.

Even if someone at some point defined "semi-direct product" for graphs, you should assume that the definition didn't stick. Keep in mind that many people studying graph theory know little about group theory.

Given two graphs $G$ and $H$ there are many different types of graph products, if you are writing a paper then you are best just to explicitly describe the edge-set outright given $G$ and $H$ and if it looks reminiscent of something from algebra, then name it appropriately.

In fact, I would even go so far to say that if you use the term in writing a paper for a research journal, that you should explicitly define what you mean by a "Cayley graph".