Implemented Smart Keyboard support for iPad Pro

[AgustinCordes]

My very first contribution to ScummVM, and I couldn't be happier! While it's a straightforward implementation, it works very well:

https://youtu.be/I3PZdIpQldI

Basically, I just had to disable the virtual keyboard on landscape orientation, but there are a few extra goodies:

- Removed the annoying iOS shortcuts toolbar for total immersion.
- Added fake ESC key support replacing the ` (thanks @donkthemagicllama for the idea from one of your posts)
- Added full support for arrow keys! This was the trickiest bit to implement but works great.

All of this should theoretically work on any Bluetooth keyboard, but for now I'm only able to test it on iPad Pro w/Smart Keyboard. I'll be pushing my branch soon.

[digitall]

AgustinCordes: Thanks for your contribution. Will look forward to seeing a Github Pull Request in future and hopefully it should get merged quickly

[dottostring]

Would you be will ing to share the changes and in what files you made them? I don't see a pull request and use an iPad Pro with Smart Keyboard. Thanks in advance!

[dottostring]

I have decided to share my solution. I couldn't wait!

In file ios7_keyboard.mm:
Change initWithKeyboard to this ->

Code: Select all

- (id)initWithKeyboard:(SoftKeyboard *)keyboard {
	self = [super initWithFrame:CGRectMake(0.0f, 0.0f, 0.0f, 0.0f)];
	softKeyboard = keyboard;
	[self setAutocorrectionType:UITextAutocorrectionTypeNo];
	[self setAutocapitalizationType:UITextAutocapitalizationTypeNone];
	[self setEnablesReturnKeyAutomatically:NO];
    UITextInputAssistantItem* item = [self inputAssistantItem];
    item.leadingBarButtonGroups = @[];
    item.trailingBarButtonGroups = @[];
	return self;
}

In file ios7_video.mm:
Change initSurface to this ->

Code: Select all

- (void)initSurface {
	if (_context) {
		[self rebuildFrameBuffer];
	}

	BOOL isLandscape = (self.bounds.size.width > self.bounds.size.height); // UIDeviceOrientationIsLandscape([[UIDevice currentDevice] orientation]);

	int screenWidth, screenHeight;
	if (isLandscape) {
		screenWidth = MAX(_renderBufferWidth, _renderBufferHeight);
		screenHeight = MIN(_renderBufferWidth, _renderBufferHeight);
	}
	else {
		screenWidth = MIN(_renderBufferWidth, _renderBufferHeight);
		screenHeight = MAX(_renderBufferWidth, _renderBufferHeight);
	}
    
    if (_keyboardView == nil) {
        _keyboardView = [[SoftKeyboard alloc] initWithFrame:CGRectZero];
        [_keyboardView setInputDelegate:self];
        [self addSubview:[_keyboardView inputView]];
        [self addSubview: _keyboardView];
        [_keyboardView showKeyboard];
    }

	glBindRenderbuffer(GL_RENDERBUFFER, _viewRenderbuffer); printOpenGLError();

	[self clearColorBuffer];

	GLfloat adjustedWidth = _videoContext.screenWidth;
	GLfloat adjustedHeight = _videoContext.screenHeight;
	if (_videoContext.asprectRatioCorrection) {
		if (_videoContext.screenWidth == 320 && _videoContext.screenHeight == 200)
			adjustedHeight = 240;
		else if (_videoContext.screenWidth == 640 && _videoContext.screenHeight == 400)
			adjustedHeight = 480;
	}

	float overlayPortraitRatio;

	if (isLandscape) {
		GLfloat gameScreenRatio = adjustedWidth / adjustedHeight;
		GLfloat screenRatio = (GLfloat)screenWidth / (GLfloat)screenHeight;

		// These are the width/height according to the portrait layout!
		int rectWidth, rectHeight;
		int xOffset, yOffset;

		if &#40;gameScreenRatio < screenRatio&#41; &#123;
			// When the game screen ratio is less than the screen ratio
			// we need to scale the width, since the game screen was higher
			// compared to the width than our output screen is.
			rectWidth = &#40;int&#41;&#40;screenHeight * gameScreenRatio&#41;;
			rectHeight = screenHeight;
			xOffset = &#40;screenWidth - rectWidth&#41; / 2;
			yOffset = 0;
		&#125; else &#123;
			// When the game screen ratio is bigger than the screen ratio
			// we need to scale the height, since the game screen was wider
			// compared to the height than our output screen is.
			rectWidth = screenWidth;
			rectHeight = &#40;int&#41;&#40;screenWidth / gameScreenRatio&#41;;
			xOffset = 0;
			yOffset = &#40;screenHeight - rectHeight&#41; / 2;
		&#125;

//        &#91;_keyboardView hideKeyboard&#93;;

		//printf&#40;"Rect&#58; %i, %i, %i, %i\n", xOffset, yOffset, rectWidth, rectHeight&#41;;
		_gameScreenRect = CGRectMake&#40;xOffset, yOffset, rectWidth, rectHeight&#41;;
		overlayPortraitRatio = 1.0f;
	&#125; else &#123;
		GLfloat ratio = adjustedHeight / adjustedWidth;
		int height = &#40;int&#41;&#40;screenWidth * ratio&#41;;
		//printf&#40;"Making rect &#40;%u, %u&#41;\n", screenWidth, height&#41;;
		_gameScreenRect = CGRectMake&#40;0, 0, screenWidth, height&#41;;

//        CGRect keyFrame = CGRectMake&#40;0.0f, 0.0f, 0.0f, 0.0f&#41;;
//        if &#40;_keyboardView == nil&#41; &#123;
//            _keyboardView = &#91;&#91;SoftKeyboard alloc&#93; initWithFrame&#58;keyFrame&#93;;
//            &#91;_keyboardView setInputDelegate&#58;self&#93;;
//            &#91;self addSubview&#58;&#91;_keyboardView inputView&#93;&#93;;
//            &#91;self addSubview&#58; _keyboardView&#93;;
//        &#125;
//
//        &#91;_keyboardView showKeyboard&#93;;
		overlayPortraitRatio = &#40;_videoContext.overlayHeight * ratio&#41; / _videoContext.overlayWidth;
	&#125;
	_overlayRect = CGRectMake&#40;0, 0, screenWidth, screenHeight * overlayPortraitRatio&#41;;

	_gameScreenCoords&#91;0&#93;.x = _gameScreenCoords&#91;2&#93;.x = CGRectGetMinX&#40;_gameScreenRect&#41;;
	_gameScreenCoords&#91;0&#93;.y = _gameScreenCoords&#91;1&#93;.y = CGRectGetMinY&#40;_gameScreenRect&#41;;
	_gameScreenCoords&#91;1&#93;.x = _gameScreenCoords&#91;3&#93;.x = CGRectGetMaxX&#40;_gameScreenRect&#41;;
	_gameScreenCoords&#91;2&#93;.y = _gameScreenCoords&#91;3&#93;.y = CGRectGetMaxY&#40;_gameScreenRect&#41;;

	_overlayCoords&#91;1&#93;.x = _overlayCoords&#91;3&#93;.x = CGRectGetMaxX&#40;_overlayRect&#41;;
	_overlayCoords&#91;2&#93;.y = _overlayCoords&#91;3&#93;.y = CGRectGetMaxY&#40;_overlayRect&#41;;

	&#91;self setViewTransformation&#93;;
	&#91;self updateMouseCursorScaling&#93;;
&#125;

Then in the same file change handleKeyPress to this ->

Code: Select all

- (void)handleKeyPress:(unichar)c {
    if (c == '`') {
        [self addEvent:InternalEvent(kInputKeyPressed, '\E', 0)];
    } else {
        [self addEvent:InternalEvent(kInputKeyPressed, c, 0)];
    }
}

Finally:
Enjoy Space Quest!

[dottostring]

Directly under SofKeyboard init function, add -->

Code: Select all

- (NSArray *)keyCommands {
    UIKeyCommand *upArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputUpArrow modifierFlags: 0 action: @selector(upArrow:)];
    UIKeyCommand *downArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputDownArrow modifierFlags: 0 action: @selector(downArrow:)];
    UIKeyCommand *leftArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputLeftArrow modifierFlags: 0 action: @selector(leftArrow:)];
    UIKeyCommand *rightArrow = [UIKeyCommand keyCommandWithInput: UIKeyInputRightArrow modifierFlags: 0 action: @selector(rightArrow:)];
    return [[NSArray alloc] initWithObjects: upArrow, downArrow, leftArrow, rightArrow, nil];
}

- (void) upArrow: (UIKeyCommand *) keyCommand {
//    [self resignFirstResponder];
    [softKeyboard handleKeyPress:0x4800];
}

- (void) downArrow: (UIKeyCommand *) keyCommand {
    [softKeyboard handleKeyPress:0x5000];
}

- (void) leftArrow: (UIKeyCommand *) keyCommand {
    [softKeyboard handleKeyPress:0x4B00];
}

- (void) rightArrow: (UIKeyCommand *) keyCommand {
    [softKeyboard handleKeyPress:0x4D00];
}

This supports the keyboard arrow!

[dottostring]

I apologize, replace the above with this for better functionality between games it appears:

- (void) upArrow: (UIKeyCommand *) keyCommand {
// [self resignFirstResponder];
[softKeyboard handleKeyPress:273];
}

- (void) downArrow: (UIKeyCommand *) keyCommand {
[softKeyboard handleKeyPress:274];
}

- (void) leftArrow: (UIKeyCommand *) keyCommand {
[softKeyboard handleKeyPress:276];
}

- (void) rightArrow: (UIKeyCommand *) keyCommand {
[softKeyboard handleKeyPress:275];
}

[basis]

Thank you so much!!!

Any chance we can get this added in to the core code so that when an update comes out I do not have to manually change this?

Thanks again.

[digitall]

I have applied the patches from these posts with minor formatting cleanup, and opened this as a Pull Request as:
https://github.com/scummvm/scummvm/pull/1225

@dottostring: If you wish to check my patch / compile / test or comment there, feel free. If you want me to amend the commit to have your github attribution, let me know.

[MWumpusZ]

I've been trying this out, and had a number of issues.


Arrow keys:

As committed and merged, arrow keys did not work for me. Possibly there was a misunderstanding about where the keyCommands function needed to be added? In any case, I moved keyCommands to iPhoneView and it works that way, PR is here: https://github.com/scummvm/scummvm/pull/1290


Virtual keyboard:

After the change, the virtual keyboard never appears, as far as I can tell. Possibly this was intentional, but surely its not a generally acceptable situation - making an external keyboard NECESSARY to play games seems like a bad idea. I haven't looked into fixing this yet.


Stops responding to keyboard input:

"Sometimes" typing doesn't end up in the games input fields. My bluetooth keyboard has a button for searching my ipad, and after pressing this and returning to scummvm, I can not enter text. Certain details strongly suggest that it's not an issue with my keyboard, and I have a few leads, but will be investigating further.


Unclear UITextAssistant stuff:

Apparently to make the build work, digitall disabled code that does... something... with a UITextInputAssistantItem, see https://github.com/scummvm/scummvm/comm ... 4e295ba305. It's not clear to me what this disabled code was supposed to do; for me, reenabling it appears to build just fine, but no effect is apparent - what is it supposed to do?


It sounds like a whole bunch of complaints, but it's great work that has been done by you guys! It looks like fixing them up shouldn't be that hard. I'll certainly be looking into various points, but any thoughts and observations - including from anyone else who has tried the changes out - would be great.

[MWumpusZ]

Incidentally, do you know how input with physical external keyboards, and virtual keyboard, and so on, is done in the android version? I have an android tablet I could try it on, but no android development environment set up

My reason for asking: It MAY be wise to have the same methods of user interaction for both android and ios, insofar as it is reasonable to do so.

[MWumpusZ]

Stops responding to keyboard input:

"Sometimes" typing doesn't end up in the games input fields. My bluetooth keyboard has a button for searching my ipad, and after pressing this and returning to scummvm, I can not enter text. Certain details strongly suggest that it's not an issue with my keyboard, and I have a few leads, but will be investigating further.

It turns out this is not entirely new; I reverted to before the merge and was able to reproduce this there:

* Connect bluetooth keyboard
* Turn to portrait mode
-> virtual keyboard DOES NOT appear (this is OK, its just an observation)
-> bluetooth keyboard input is now "accepted"
* Press search ipad button
* Cancel
-> bluetooth keyboard input is NOT accepted

Rotating to landscape and back makes it works again, so evidently that action "resets" something relevant.

In any case, this is evidently a separate bug.

[MWumpusZ]

Unclear UITextAssistant stuff:

Apparently to make the build work, digitall disabled code that does... something... with a UITextInputAssistantItem, see https://github.com/scummvm/scummvm/comm ... 4e295ba305. It's not clear to me what this disabled code was supposed to do; for me, reenabling it appears to build just fine, but no effect is apparent - what is it supposed to do?

Evidently it hides the "minimised" keyboard bar that appears when the virtual keyboard is hidden. For some reason on my physical 2018 iPad (6th generation) this does not appear regardless of whether or not this code is included, but when simulating the same device it does, and adding/removing this code has the expected effect. Wut?

[digitall]

MWumpusZ : Thanks for taking a look at this code and seeing about fixing it up. I am not an iOS expert and don't even own any Apple or iOS devices, so could not test this, but though it best to get the code into the tree and worked on. I tried to fix it up as far as possible where logical, but I am afraid I am not an expert here so can't offer more guidance / knowledge.

[MWumpusZ]

Unclear UITextAssistant stuff:

Evidently it hides the "minimised" keyboard bar that appears when the virtual keyboard is hidden. For some reason on my physical 2018 iPad (6th generation) this does not appear regardless of whether or not this code is included, but when simulating the same device it does ...

Whether or not this bar appears at all is evidently (also) influenced by the keyboard configuration in the system settings. Fun!