I don’t see it on the site, but would it be safe to assume the Mad Catz Arcade FightStick Versus SH for 360 is around the same quality as the other MCZ sticks? Also, do the Real Arcade Pro VX SA and Real Arcade Pro VX SA KAI share the same PCB?
It doesn’t matter what I hold on my PS360+. I can’t get mine to say anything other than unknown device.
So based on Undamned’s explain action of the tests, these results can vary per console and per game?
I agree with this.
Haha. I would like to see you do some repeat testing as well. Especially since you’d also be able to test XB1/PS4 standard controllers (obviously have to use a different game) due to you padhacking often and random sticks coming in fairly often that Teyah might not have easy access to.
For the folks that feel their ps360+ is useless now because of these results, hit me up to send it my way. 
Would be nice to see the tests of MC Cthulhu with both 1 and 10 ms firmware.
Yes, they can and do (totally different console hardware, SFIV does not use GG’s program code, etc.). Before I get into some alt. test data, I need to address a few things. @Teyah, I finally got around to reading your page which details your testing. From your Preface:
I have no problem with this text. In fact, this is exactly what you should be testing for. This is not what you are testing for. You are testing much more than what is stated in your preface. You are testing (Time Segment 1) the time from the button press to the time at which the PCB sends the input change to the game console to (Time Segment 2) the time at which the game program sees the input change to (Time Segment 3) the time at which the game program does something with the input change. Only Time Segment 1 (what you listed in your preface) is in the hands of the controller manufacturers. Based on your own words ("[letter rankings] reaffirm the importance of these results, and may just influence the manufacturers to take notice and work to improve their products."), you are holding controller manufacturers’ accountable for all 3 Time Segments by lumping them together in your test results.
Also, you must realize that not many people will read/understand your test methodology/results. When you list a “0ms” response time on your sheet, what do you think most people will read that as? “0ms” perhaps? We both know it’s not 0ms, nothing is, so if it is actually the response time of your “control” controller (which you do state on your site, which most people won’t bother reading), you should list the result as “X + 0ms” and then at the bottom of your list put “X = response time of Controller Z.” That simple adjustment right there brings your test more credibility (still not credible for the reasons mentioned above, but moving on) with people who care about good data, and more importantly/likely, gives people who don’t understand the data some clue that there is more to this list than what they catch at first glance. And to make your testing even more transparent, I would add to the bottom of the list the game which was used for testing. If you want to back any claims that your data is 100% transferable to all games on both consoles (and what about PC?), you will be a busy boy.
With that out of the way, I did some testing recently on my UD-CPS2 host. My test is not perfect, either, but it’s way closer to perfect than any console based test. The only variable in my test is the amount of time it takes my USB host board to process the message read from the controller and activate the output tied to the arcade board. Based on my testing, the processing time of my USB host can be as fast as 0.1ms, so, keep that in mind when questioning how much of an effect it has on these test results. This is my setup:
[button]->[controller]->[USB Host]->[JAMMA edge]
Time Reference: Absolute (0ms is actually 0ms; shocker!)
Here is a stored image from the oscilloscope:
Channel 1 (Yellow) is the button press (active low). Channel 2 (Blue) is the button output signal (active low) of the USB host (the button input to the JAMMA edge of the CPS2 motherboard).
Test Results:
PS360+ (Firmware Rev. 1.2) - PS3 Mode: Worst Sample = 2.1ms, Best Sample = 1.1ms, 100+ Sample average = 1.6ms
PS360+ (Firmware Rev. 1.2) - 360 Mode: Worst Sample = 6.9ms, Best Sample = 2.2ms, 100+ Sample average = 4.5ms
Madcatz Fightstick PRO PS3: Worst Sample = 18ms, Best Sample = 9.4ms, 100+ Sample average = 13.8ms
-ud
Wouldn’t it be easier if each controller manufacturer showed this data - like how many ms have the PCB? and other details - in your technical specifications at your websites, before we buy it?
godlike, thanks undamned
I do need to update the site overview to specifically state that I am testing for the relative delay of arcade sticks relative to the control arcade stick tested, yes.
Time Segments 2 and 3 in your definition should not be biased towards any one brand or PCB - ie. GGXX isn’t going to identify a Mad Catz PCB and say “hey, it’s Mad Catz - I need to add another half frame of delay here!”. There should be no net effect favouring one brand or another from the game itself. I tested 500 trials over the weekend on both SF4 and GGXX with a Mad Catz TE, with no major differences noted. I may do a full 1,000 trials and retest a couple of the new sticks that are being tested, but I would be very surprised to see any significant differences. So this point seems to be more of a matter of semantics, rather than something I need to account for and correct in my testing methods.
Regarding your 2nd point, in my last site update I changed my table to list delay in +ms, so now results are listed as +X ms instead of just X ms.
Finally, regarding your oscilloscope test findings. Those are useful for finding the raw delay of arcade sticks under optimal conditions (which are very different than actual PS3 and XBox 360 conditions). It seems like you’re expecting this project of mine to be something different than what I’ve presented it as.
The intent of my project is to find the (relative) delay of different arcade sticks when used on their respective console. That is, the delay of a specific model of PS3 arcade stick while playing a PS3 game, and the delay of a specific model of XBox 360 arcade stick while playing a XBox 360 game. Your oscilloscope test findings on your custom board serve a very different purpose, so I can’t say that your results are useful in determining the actual, final delay that is experienced by console players. It can make for a good comparative set of data, but since your own custom setup doesn’t account for the console’s specifications and is polling sticks at a higher frequency than the PS3 / XBox 360, I think it’s fair to say that console players won’t find it very useful to read that data.
Except as he already pointed out different games could lag differently and there are a number of things that could change the results because of the input processing chain. Also, his data is perfectly fair because it actually tests the board itself and people can make a real decision on that. If a board tests fine but is laggier on the game tested on, then that is not the fault of the board which is where your data points to rather than other parts of the processing chain. Also, when it comes to PS4, if a driver is written shitty for a board, then that is not the fault of the board.
This is why the test should have been raw from the PCB in the first place.
I already addressed lag from game to game above in my post.
His data is perfectly fair, but not particularly useful to find the end result of console delay. Which is the point of my testing. At the end of the day what really matters for the great majority of players (console players) is the end result of the input delay experienced on PS3/360.
Both types of data are useful and if one was missing someone would want to gather the other.
Raw PCB response tells us what sort of processing delay is inherent to a PCB. Console in-game response gives a real world result. Any unexpected discrepancies between the two would be something else to look at.
Your results page is still misleading. What do you think people draw from this line of text:
“S Tier: Hori Real Arcade Pro VX-SA (+0.00 ms - “lagless” control stick)”
Perhaps that the Hori Real Arcade Pro VX-SA is lagless? How could you expect them to think otherwise? If you want people to understand your data, you need to be explicit. You could ensure less confusion by simply adding a single line to the top of your tables, something like:
“All Response times below are referenced to that of the Hori Real Arcade Pro VX-SA”
I’m expecting this project to line up with the Preface on your site.
What you just wrote perfectly captures what you are actually testing. You are indeed testing the entire scenario. That is definitely NOT what is conveyed in your Preface. If you made explicit your time references and changed the Preface on your site (and where ever else you’ve shared this project) to reflect very closely what you just wrote above, my greatest concerns with your project are taken care of. Without those changes, the data is grossly misleading.
-ud
I’m aware that the Overview needs updating to be more explicit, and that will be addressed on the next site updateI
I dont see any issue with stating that the Hori VX-SA on 360, which ranks at +0.00 ms, is “lagless”. (The quotation marks denote a soft, colloquial usage of the term which fits with the results that are presented)
Science/tech oriented people will automatically understand that the “lagless” rating is not absolute but it’s probably in the best interests of the majority of users, and the objectivity of the results/data, to clearly state that the VX-SA is the baseline from which the other sticks are measured.
Looking forward to it.
It is only lagless relative to itself! You still have no idea how much time it takes your reference sticks to respond to a button press (what your current Preface claims is the point of your entire study).
-ud
I was. Thanks for that update!!!
Just adding some visual context to those who are threading this thread
Navy Officer and computer pioneer Grace Hopper shows her Nanosecond of wire
https://www.youtube.com/watch?v=JEpsKnWZrJ8
In the sense that it’s an absolute, I don’t think that that lagless an approprite description at all. (It’s akin to the people who say that any lag less than 1 frame doesn’t matter.) What happens if you find a stick that is faster than the Hori VS-SA? Maybe you’ll say that that stick breaks the laws of causality to offer negative lag.
I guess I understand that it’s not supposed to be absolute, in so far as I think it’s an abuse of the term.