Meaning of quote from 'God Emperor of Dune'

Solution 1:

Leto is insulting Luyseyal's capacity to distinguish an automaton (a machine) from an intelligent being.

A yardstick is a tool of measure. "Your yardsticks" alludes to his conversational partner's ability to measure, or more generally to assess, how something else functions.

That "something else," contextually, is an automated machine (the "Ixians' machine" on the page before), and the conversation has turned to whether whether it is intelligent or merely an automaton. The speaker, the God-Emperor Leto, taunts the Bene Gesserit listeners with that possibility earlier in the passage by posing this question:

"Is automation synonymous with conscious intelligence?" he asked.

One of the Bene Gesserit, Anteac, distinguishes between intelligence and automation:

"The machines cannot anticipate every problem of importance to humans. It is the difference between serial bits and an unbroken continuum. We have the one; machines are confined to the other."

Leto seems to approve. However, the other Bene Gesserit, Luyseyal, does not get it and demands an explanation.

"Share!" Luyseyal said.

The quote you ask about (Leto now speaking to Luyseyal) reiterates the distinction - if the incredulous Bene Gesserit Luyseyal cannot assess the Ixian machine, she thinks she is engaged with intelligence, not automation. This repeats one interpretation of the Turing test: that if a human tester cannot assess the difference between a human respondent and a machine respondent, then the machine is no longer distinguishable from human intelligence. Leto is putting Luyseyal back in her place by suggesting the machine is beyond her measure both literally (Luyseyal thinks it's intelligent, but it isn't) and figuratively (Luyseyal thinks she is more intelligent than she is; her yardstick, or capacity for measure, isn't as long as Anteac's, who can still explain the difference between the machine and a human).

Whether Luyseyal catches all of that or not, she at least notices that she was insulted:

Luyseyal was angry now and not bothering to conceal it.

One detail is that the statement was said as a statement of fact, but I've treated it differently. I repeat it one more time with slightly more context to emphasize that it's a statement of fact for Luyseyal alone:

Leto looks at the puzzled Leyseyal. "If it falls outside your yardsticks, then you are engaged with intelligence, not with automation," he said.

Leto, the speaker, is a gnomic character who in this scene and others plays with suggestions and ambiguity. He is addressing Luyseyal in particular, and every "you" should be read referring to her rather than to all people. He's spitting Luyseyal's logic back at her while insulting her at the same time.