Re - Request/Response
Ṽ - Validation
ReVoman is an API automation tool for JVM (Java/Kotlin) from the API-first SaaS company, Salesforce. It re-imagines API automation by letting you execute a Postman collection in a JVM program/test.
To start with, think of it as Postman for JVM (Java/Kotlin); that emulates this Run button on a collection through a Java program. But it’s even better
It strikes a balance between flexibility provided by low-level tools like REST Assured and ease of use provided by UI tools like Postman
Maven
<dependency>
<groupId>com.salesforce.revoman</groupId>
<artifactId>revoman</artifactId>
<version>0.21.7</version>
</dependency>Bazel
"com.salesforce.revoman:revoman"Gradle Kts
implementation("com.salesforce.revoman:revoman:0.21.7")-
The majority of JVM SaaS applications are REST-based. But the API automation is done through a Mul-T-verse of Integration/Functional tests, E2E tests and Manual tests, each with its own frameworks, tools, and internal utilities, testing almost the same code flow.
-
These custom alien automation frameworks, often built using low-level tools like REST Assured which are specific to a service or domain and are rigid to reuse, extend and difficult to maintain.
-
This automation competes on cognitive complexity and learning curve with the Prod code, and mostly, automation wins.
-
After a point, the API automation may deviate from its purpose of augmenting real end-user interaction and turns into a foot-chain for development.
Contrary to these custom frameworks, almost every team uses Postman for manual testing their APIs. Postman collections contain a lot of information about your APIs and the order in which they need to be executed for manual testing, in a Structured Template. Leveraging it, can mitigate writing a lot of code as we translate those manual steps into automation.
How productive would it be, if you can plug your exported Postman collection, that you anyway would have created for your manual testing, and execute them through your JVM tests?
How about a Universal API automation tool promotes low-code and low-cognitive-complexity and strikes a balance between flexibility and ease of use?
Let’s check out how you can perform Template-Driven-Testing:
Invoke revUp, supplying your postman templates and environments, along with your customizations through a configuration, to kick-off the execution
ReVoman.revUp(
Kick.configure()
...
.off())Let’s check out how to build this config with an example:
ReVoman.revUp( // (1)
Kick.configure()
.templatePaths(PQ_TEMPLATE_PATHS) // (2)
.environmentPath("pm-templates/pq/pq-env.postman_environment.json") // (3)
.dynamicEnvironment( // (4)
Map.of(
"$quoteFieldsToQuery", "LineItemCount, CalculationStatus",
"$qliFieldsToQuery", "Id, Product2Id",
"$qlrFieldsToQuery", "Id, QuoteId, MainQuoteLineId, AssociatedQuoteLineId"))
.customDynamicVariable( // (5)
"$quantity", ignore -> String.valueOf(Random.Default.nextInt(10) + 1))
.haltOnAnyFailureExceptForSteps(STEPS_TO_IGNORE_FOR_FAILURE) // (6)
.requestConfig( // (7)
unmarshallRequest(
ASYNC_STEP_NAMES,
PlaceQuoteInputRepresentation.class,
adapter(PlaceQuoteInputRepresentation.class)))
.hooks( // (8)
pre(
ASYNC_STEP_NAMES,
(stepName, requestInfo, rundown) -> {
final var pqInputRep =
requestInfo.<PlaceQuoteInputRepresentation>getTypedTxObj();
assertThat(pqInputRep).isNotNull();
if ("pq-create: qli+qlr (skip-pricing)"
.equals(pqInputRep.getGraph().getGraphId())) {
LOGGER.info("Skip pricing for step: {}", stepName);
rundown.mutableEnv.set("$pricingPref", PricingPref.Skip.toString());
} else {
rundown.mutableEnv.set("$pricingPref", PricingPref.System.toString());
}}),
post("query-quote-and-related-records", PQE2ETest::assertAfterPQCreate),
post(
ASYNC_STEP_NAMES,
(stepName, rundown) -> {
LOGGER.info(
"Waiting after Step: {} for the Quote: {} to get processed",
stepName,
rundown.mutableEnv.getString("quoteId"));
// ! CAUTION 10/09/23 gopala.akshintala: This test can be flaky until
// polling is implemented
Thread.sleep(20000);}))
.responseConfig( // (9)
unmarshallSuccessResponse("quote-related-records", CompositeResponse.class), // <9.1>
validateIfSuccess( // <9.2>
ASYNC_STEP_NAMES,
PlaceQuoteOutputRepresentation.class,
validatePQSuccessResponse),
validateIfFailed(
FAILURE_STEP_NAMES,
PlaceQuoteOutputRepresentation.class,
validatePQErrorResponse))
.insecureHttp(true) // (10)
.off()); // Kick-off-
revUpis the method to call passing a configuration, built as below -
Supply the path (relative to resources) to the Template Collection JSON file
-
Supply the path (relative to resources) to the Environment JSON file
-
Supply any dynamic environment that is runtime-specific
-
Plug Custom dynamic variables that are prepared at runtime
-
It lets you be in charge of the Step Orchestration by letting you configure a pass-list of steps to ignore for failure
-
You can provide a strong type for your request to be marshalled into. You can use this strong type while executing pre-hooks
-
You can set up a pre- / post-hook around a Step via
hooks, which can help as callbacks, especially for Async operations.-
These runs despite the step are successful or failed. The entire rundown till that step shall be provided to the hook as a parameter
-
-
Provide configuration to unmarshall/deserialize response JSON into strong types <9.1>
unmarshallSuccessResponseandunmarshallErrorResponsehelp you use strong types inside hooks. <9.2> Supply your validations/assertions to be run on a step response throughvalidateIfSuccessandvalidateIfFailed-
You can leverage the power of Vador to write config-driven validations
-
-
[Not for Prod] Ignore Java cert issues when firing Http calls
After all this, you receive back a detailed Rundown of all the steps with all the Request-Response data, along with all Strong types that you configured, such that you can seamlessly run more assertions on top of the run.
Rundown(
stepNameToReport: Map<String, StepReport>,
environment: PostmanEnvironment)
StepReport(
val status: String,
val requestInfo: Either<RequestFailure, TxInfo<Request>>?,
val preHookFailure: PreHookFailure?,
val responseInfo: Either<ResponseFailure, TxInfo<Response>>?,
val postHookFailure: PostHookFailure?,
val envSnapshot: PostmanEnvironment<Any?>
)It captures failure at any stage of a step during execution. Here is what all can go wrong:
This rundown has everything you need to know about what happened during each step execution, along with environment snapshot during that step execution. Here is what a debugger view of a Rundown looks like:
Let’s zoom into a detailed view of one of those Step reports:
Here are the environment key-value pairs accumulated along the entire execution:
Rundown has many convenience methods to ease applying further assertions on top of it.
-
ReVoman internally uses a modern JSON library called Moshi
-
There may be a POJO that inherits or contains legacy classes which are hard or impossible to serialize. ReVoman lets you serialize such a legacy POJO by letting you pass
typesToIgnoreForMarshalling, where you can you filter-out these legacy classes. -
The payload may not map to POJO, and you may need a custom adapter for Conversion. Moshi has it covered for you with its advanced adapter mechanism and ReVoman accepts Moshi adapters via
requestConfig,responseConfigandcustomAdapters -
ReVoman also comes bundled with JSON reader utils and JSON writer utils to help build Moshi adapters.
|
💡
|
Refer ConnectInputRepWithGraphAdapter for an advanced adapter use-case |
-
A hook lets you fiddle with the execution by plugging in your code before or after a Step execution
-
You get the stepName and the entire rundown of what happened prior to this step as parameters, as demonstrated in PQE2ETest.java
.hooks(
pre(
stepName(s)ForHook,
(currentStepName, requestInfo, rundown) -> {
//...callback-code...
}),
post(
stepName(s)ForHook,
(currentStepName, rundown) -> {
//...callback-code...
})
)-
You can do things like assertion on the rundown or environment and even mutate the environment, such that the execution of subsequent steps picks up those changes.
|
💡
|
Here is a low-code E2E test |
Compared to a traditional Integration/Functional or E2E test, approximately, the amount of code needed is 89% less using ReVoman. The above test doesn’t just have low-code, but also low in Cognitive Complexity and Transparent in what it does.
-
ReVoman is like a library that you can plug into any JVM program/test. Apart from adding a dependency in the build tool, there is no extra setup needed to hook these tests into CI/CD.
-
A nice side effect is, this lets the Postman collections always stay up to date and the entire Postman collection guards each check-in in the form of a Test suite augmenting manual testing.
-
Any day, you can find an up-to-date Postman collection for every feature you need to test, right in your VCS (Git) along with your code. Developers can import these templates directly from VCS into Postman for manual testing. This comes in very handy during a team blitz.
-
It reads the environment JSON provided into in-memory.
-
Then it reads and Inflates each static template in the collection, replacing variables at runtime from the in-memory environment
-
It unmarshalls the request into Strong JVM types using
requestConfigand executes the pre-hooks for the step -
It then does an HTTP request.
-
It reads the response and executes Postscript JS on the response and updates the in-memory environment.
-
It unmarshalls the response into Strong JVM types using the
responseConfigsupplied and lets you run Type-safe validations on the strong-type and fails-fast at first failure. -
Finally, all the post-hooks for the step get triggered
-
The iteration continues for all the steps in the template collection
-
Bring a Unified & Simplified Test strategy across the mul-T-verse (FTests, E2E Tests, and Manual testing with Postman) for any service. This is a generic tool, and just by changing the template, the same config/pattern can be reused for any feature flow agnostic of it being an FTest or E2E test
-
Transparency and better Traceability of issues
-
This forces engineers to think like API-first customers while writing tests.
-
FTest Data setup: You can use the ReVoman for the FTest data setup too. This eliminates the need for different teams to write their own internal utilities for data setup.
-
E2E Test can even reside outside the Service repo, as long as it can hit the service
The future looks bright
-
API metrics and Analytics
-
It’s built with extensibility in mind. It can easily be extended to support other template formats.
-
You should be able to run Kaiju availability tests right from your IDE and debug them too
-
-
In-built polling support for Async steps
-
Payload generation
-
Flow control through YAML config
-
IDE debug points in the Prod code work as expected while running the test.
-
Coming to FTest code, we debug when we don’t understand what’s going on in the code.
-
Debugging necessarily doesn’t have to be with a debug point in the IDE.
-
To be able to debug, a developer needs to be informed about what went wrong and he/she should have ways to try and test an isolated portion of the run.
-
In the case of ReVoman, you have the whole Postman collection at your disposal along with the Rundown. The entire test is transparent.
-
This experience can be enhanced with more logging and better assertions.
Why not use Newman or Postman CLI?
-
ReVoman may be similar to Newman or Postman CLI when it comes to executing a Postman collection, but the similarities end there.
-
Newman/Postman CLI are built for node cannot be executed within a JVM. Even if you are able to run with some hacky way, there is no easy way to assert results.
-
ReVoman is JVM first that lets you configure a lot more, and gives you back a detailed report to assert in a typesafe way
-
Newman is limited and cannot be integrated into our automation model on JVM