-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a possibility to add error injections to the code from system tests #2948
Comments
I don't think we can do this in a generic way, too many possible errors. Likely it'd also be some kind of control service feature since it's the node that should do this, not CLI. But for your specific use case we can make some kind of multipart upload via CLI which will push a part of the object, but not finish it. |
We can start with one error location and one specific error for my case, that cause the system to create an unfinished object. And then extend these errors when it is needed, we don't need to cover every possible situation at once. Ideally, every service should implement this type of functionality. It doesn't matter how to invoke it, either via cli, or rest, or by specifying env variable. At least we will have a generic/code syntax to cause it. |
There are good things in error injection, don't get me wrong, I'm just not sure how we can implement it better. Error injection can really simplify testing some hardly reproducible things like disk errors or specific network interaction problems. At the same time there is some cost to it, it can get deep into the code (including hot paths of various kind) and control service will get this super-ability that could be abused. |
@evgeniiz321, do you have some understanding of what you want as a new test framework? Is But again, only if we know that we need it and there is no external request for it. For the only provided wish in this issue, we can just solve #952. It appeared years ago and tests were not the original requester. I can imagine that it may be helpful in general. |
The problem is, sometimes you need to break the node. |
@carpawell details of the implementation is on you, from tests it is ok to work with rest api, with a new cli tool (or old one), with env variables, with any other possible variations, you should pick the one that is easiest for you to implement. I'm not sure it can be done via a separate internal thing, but if it can be done - that is fine to me. |
these are lightweight scripts to me, they may be written in Go, and not a part of the neofs-cli. I'd maintain them in neofs-testcases repo, and the best - nspcc-dev/neofs-testcases#258 |
nspcc-dev/neofs-testcases#258 won't help. And scripts won't help as well. If you need some weird behavior from node, you have to have some code in node to do that. |
so its not about CLI overriding but the node. Veeery strange to have this in the "main" node, it's a custom node to me |
@carpawell, think of testing our relay/upload problem. How can we do that without a node purposefully failing requests? How can we get a node with this behavior? |
Do you want to solve it in this issue's context? Well, we need some status response that does not stop a non-container node. This may work: https://github.com/nspcc-dev/neofs-api/blob/d7c033317a66aff467728f4ca5a86879edb66e54/status/types.proto#L116-L123. Put lock when it is not allowed or PUT TS on a lock object. Putting already removed objects also could work. |
For tests like these - nspcc-dev/neofs-testcases#794 we need a way to put the system under the test to certain conditions that can't be easily and reliably reproduced under regular circumstances, but we can encounter them running the production code.
The most convenient thing from the test perspective would be the possibility to add error injections to some code places. E.g. if we want to get an unfinished object and want to validate how it is cleaned, it would be nice to inject an error during the object creation operation and got such an unfinished object.
Same feature could be also applied to verify more unusual scenarios and different states of the system. To ensure the robust recovery from other possible faulty situations.
I would expect it to be implemented either via env variables/config files before service starts, or, more conveniently via some internal cli commands - like
./neofs-cli inject-error ERROR_LOCATION ERROR_TYPE
, where ERROR_LOCATION is a specific place in the code and ERROR_TYPE is either a panic, or a sleep with different number of seconds, or a bool value that somehow is used in the code. E.g../neofs-cli inject-error OBJECT_CREATION_POST_PARTS_CREATE RETURN_FALSE
The text was updated successfully, but these errors were encountered: