A conversation very much worth having, however either of us (or anyone else) thinks it comes out (for the present). Yes, there are of course issues of how much activity there is in the brain at any given time (see Shoham et al. "How silent is the brain?" J Comp Physiol A, 2006). Most neurons don't, as you say, generate action potentials all the time at maxium frequency. On the flip side, though, there is lots of relevant signalling/processing going on in neurons without any action potential generation at all (of the five cell types in the retina only ganglion cells actually generate action potentials). My estimate was "for fun" but I still think adequate to show why Moravec's was flawed.

They key issues are, in any case, actually not here but in the architectual realm and, as you say, in the "what if" realm. And there I'm a little less concerned about the latter than you are. We actually do already and every day create large numbers of emergent "intelligences" (we call them "babies"). And they do indeed create a variety of problems and hazards. But we also have several millenia at least of experience in working with such unpredictable machines and so know a fair amount both about their benefits and about how to guard against the associated risks.

You (and others) might be amused by efforts to estimate the computing power not only of brains but of the universe. See If the Universe Were a Computer. Interesting to think about whether these estimates do/do not have some of the same problems as Moravec.